Monitoring and self-healing of deployed environments

ABSTRACT

In various examples, a system identifies a first issue object associated with the alert by making a first set of determinations, based on an alert of an active issue of a system resource. Additionally, the system can determine whether the active issue associated with the first issue object can be automatically corrected by one or more self-healing processes, based on the first issue object. Moreover, the system can implement the one or more self-healing processes, based on determining that the active issue associated with the first issue object can be automatically corrected by one or more self-healing processes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/693,139 filed Nov. 22, 2019; the aforementioned priority applicationbeing hereby fully incorporated by reference in their entirety.

TECHNICAL FIELD

This application relates generally to monitoring of deployedenvironments and, more particularly, to automatically monitoring andcorrecting deployed environments.

BACKGROUND

Monitoring of deployed environments includes evaluating, monitoring, andmanaging deployed services, applications, systems, databases, or otherinfrastructure (referred to herein as “components” or “resources”) toensure proper operation and health. In cloud-based environments withmultiple instances of a resource, monitoring may be performed at variousgranularity, including monitoring individual instances, individualresources, clusters, and/or any other granularity.

Cloud monitoring allows a system administrator or other individual tocorrect issues in the deployed environment, such as unresponsiveresources, incorrect deployments, and other problems. Current systemsrequire users to interact with monitoring services to determine thecurrent status of an environment. If a problem is detected, a systemadministrator or other individual must manually correct the error, forexample, by redeploying one or more resources.

SUMMARY

In various embodiments, a system for monitoring a cloud environment isdisclosed. The system includes a computing device configured to receivean alert associated with an active issue related to a component,generate an issue object based on the alert, and execute a self-healingflow based on an alert type of the alert.

In various embodiments, a non-transitory computer readable medium havinginstructions stored thereon is disclosed. The instructions, whenexecuted by a processor cause a device to perform operations includingreceiving an alert associated with an active issue related to acomponent, generating an issue object based on the alert, and executinga self-healing flow based on an alert type of the alert.

In various embodiments, a computer-implemented method is disclosed. Themethod includes steps of receiving an alert associated with an activeissue related to a component, generating an issue object based on thealert, and executing a self-healing flow based on an alert type of thealert.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the present invention will be more fullydisclosed in, or rendered obvious by the following detailed descriptionof the preferred embodiments, which are to be considered together withthe accompanying drawings wherein like numbers refer to like parts andfurther wherein:

FIG. 1 illustrates a block diagram of a computer system, in accordancewith some embodiments.

FIG. 2 illustrates a network configured to monitor and maintain a cloudenvironment, in accordance with some embodiments.

FIG. 3 illustrates a monitoring process, in accordance with someembodiments.

FIG. 4 illustrates a self-healing and user-alerting process for anactive issue, in accordance with some embodiments.

FIG. 5 illustrates an automatic correction process for an active issue,in accordance with some embodiments.

FIG. 6 illustrates a load balancing process for a received alert, inaccordance with some embodiments.

DETAILED DESCRIPTION

The description of the preferred embodiments is intended to be read inconnection with the accompanying drawings, which are to be consideredpart of the entire written description of this invention. The drawingfigures are not necessarily to scale and certain features of theinvention may be shown exaggerated in scale or in somewhat schematicform in the interest of clarity and conciseness. In this description,relative terms such as “horizontal,” “vertical,” “up,” “down,” “top,”“bottom,” as well as derivatives thereof (e.g., “horizontally,”“downwardly,” “upwardly,” etc.) should be construed to refer to theorientation as then described or as shown in the drawing figure underdiscussion. These relative terms are for convenience of description andnormally are not intended to require a particular orientation. Termsincluding “inwardly” versus “outwardly,” “longitudinal” versus “lateral”and the like are to be interpreted relative to one another or relativeto an axis of elongation, or an axis or center of rotation, asappropriate. Terms concerning attachments, coupling and the like, suchas “connected” and “interconnected,” refer to a relationship whereinstructures are secured or attached to one another either directly orindirectly through intervening structures, as well as both moveable orrigid attachments or relationships, unless expressly describedotherwise. The term “operatively coupled” is such an attachment,coupling, or connection that allows the pertinent structures to operateas intended by virtue of that relationship. In the claims,means-plus-function clauses, if used, are intended to cover structuresdescribed, suggested, or rendered obvious by the written description ordrawings for performing the recited function, including not onlystructure equivalents but also equivalent structures.

FIG. 1 illustrates a computer system configured to implement one or moreprocesses, in accordance with some embodiments. The system 2 is arepresentative device and may comprise a processor subsystem 4, aninput/output subsystem 6, a memory subsystem 8, a communicationsinterface 10, and a system bus 12. In some embodiments, one or more thanone of the system 2 components may be combined or omitted such as, forexample, not including an input/output subsystem 6. In some embodiments,the system 2 may comprise other components not combined or comprised inthose shown in FIG. 1 . For example, the system 2 may also include, forexample, a power subsystem. In other embodiments, the system 2 mayinclude several instances of the components shown in FIG. 1 . Forexample, the system 2 may include multiple memory subsystems 8. For thesake of conciseness and clarity, and not limitation, one of each of thecomponents is shown in FIG. 1 .

The processor subsystem 4 may include any processing circuitry operativeto control the operations and performance of the system 2. In variousaspects, the processor subsystem 4 may be implemented as a generalpurpose processor, a chip multiprocessor (CMP), a dedicated processor,an embedded processor, a digital signal processor (DSP), a networkprocessor, an input/output (I/O) processor, a media access control (MAC)processor, a radio baseband processor, a co-processor, a microprocessorsuch as a complex instruction set computer (CISC) microprocessor, areduced instruction set computing (RISC) microprocessor, and/or a verylong instruction word (VLIW) microprocessor, or other processing device.The processor subsystem 4 also may be implemented by a controller, amicrocontroller, an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a programmable logic device (PLD),and so forth.

In various aspects, the processor subsystem 4 may be arranged to run anoperating system (OS) and various applications. Examples of an OScomprise, for example, operating systems generally known under the tradename of Apple OS, Microsoft Windows OS, Android OS, Linux OS, and anyother proprietary or open source OS. Examples of applications comprise,for example, network applications, local applications, data input/outputapplications, user interaction applications, etc.

In some embodiments, the system 2 may comprise a system bus 12 thatcouples various system components including the processing subsystem 4,the input/output subsystem 6, and the memory subsystem 8. The system bus12 can be any of several types of bus structure(s) including a memorybus or memory controller, a peripheral bus or external bus, and/or alocal bus using any variety of available bus architectures including,but not limited to, 9-bit bus, Industrial Standard Architecture (ISA),Micro-Channel Architecture (MSA), Extended ISA (EISA), Intelligent DriveElectronics (IDE), VESA Local Bus (VLB), Peripheral ComponentInterconnect Card International Association Bus (PCMCIA), SmallComputers Interface (SCSI) or other proprietary bus, or any custom bussuitable for computing device applications.

In some embodiments, the input/output subsystem 6 may include anysuitable mechanism or component to enable a user to provide input tosystem 2 and the system 2 to provide output to the user. For example,the input/output subsystem 6 may include any suitable input mechanism,including but not limited to, a button, keypad, keyboard, click wheel,touch screen, motion sensor, microphone, camera, etc.

In some embodiments, the input/output subsystem 6 may include a visualperipheral output device for providing a display visible to the user.For example, the visual peripheral output device may include a screensuch as, for example, a Liquid Crystal Display (LCD) screen. As anotherexample, the visual peripheral output device may include a movabledisplay or projecting system for providing a display of content on asurface remote from the system 2. In some embodiments, the visualperipheral output device can include a coder/decoder, also known asCodecs, to convert digital media data into analog signals. For example,the visual peripheral output device may include video Codecs, audioCodecs, or any other suitable type of Codec.

The visual peripheral output device may include display drivers,circuitry for driving display drivers, or both. The visual peripheraloutput device may be operative to display content under the direction ofthe processor subsystem 6. For example, the visual peripheral outputdevice may be able to play media playback information, applicationscreens for application implemented on the system 2, informationregarding ongoing communications operations, information regardingincoming communications requests, or device operation screens, to nameonly a few.

In some embodiments, the communications interface 10 may include anysuitable hardware, software, or combination of hardware and softwarethat is capable of coupling the system 2 to one or more networks and/oradditional devices. The communications interface 10 may be arranged tooperate with any suitable technique for controlling information signalsusing a desired set of communications protocols, services or operatingprocedures. The communications interface 10 may comprise the appropriatephysical connectors to connect with a corresponding communicationsmedium, whether wired or wireless.

Vehicles of communication comprise a network. In various aspects, thenetwork may comprise local area networks (LAN) as well as wide areanetworks (WAN) including without limitation Internet, wired channels,wireless channels, communication devices including telephones,computers, wire, radio, optical or other electromagnetic channels, andcombinations thereof, including other devices and/or components capableof/associated with communicating data. For example, the communicationenvironments comprise in-body communications, various devices, andvarious modes of communications such as wireless communications, wiredcommunications, and combinations of the same.

Wireless communication modes comprise any mode of communication betweenpoints (e.g., nodes) that utilize, at least in part, wireless technologyincluding various protocols and combinations of protocols associatedwith wireless transmission, data, and devices. The points comprise, forexample, wireless devices such as wireless headsets, audio andmultimedia devices and equipment, such as audio players and multimediaplayers, telephones, including mobile telephones and cordlesstelephones, and computers and computer-related devices and components,such as printers, network-connected machinery, and/or any other suitabledevice or third-party device.

Wired communication modes comprise any mode of communication betweenpoints that utilize wired technology including various protocols andcombinations of protocols associated with wired transmission, data, anddevices. The points comprise, for example, devices such as audio andmultimedia devices and equipment, such as audio players and multimediaplayers, telephones, including mobile telephones and cordlesstelephones, and computers and computer-related devices and components,such as printers, network-connected machinery, and/or any other suitabledevice or third-party device. In various implementations, the wiredcommunication modules may communicate in accordance with a number ofwired protocols. Examples of wired protocols may comprise UniversalSerial Bus (USB) communication, RS-232, RS-422, RS-423, RS-485 serialprotocols, FireWire, Ethernet, Fibre Channel, MIDI, ATA, Serial ATA, PCIExpress, T-1 (and variants), Industry Standard Architecture (ISA)parallel communication, Small Computer System Interface (SCSI)communication, or Peripheral Component Interconnect (PCI) communication,to name only a few examples.

Accordingly, in various aspects, the communications interface 10 maycomprise one or more interfaces such as, for example, a wirelesscommunications interface, a wired communications interface, a networkinterface, a transmit interface, a receive interface, a media interface,a system interface, a component interface, a switching interface, a chipinterface, a controller, and so forth. When implemented by a wirelessdevice or within wireless system, for example, the communicationsinterface 10 may comprise a wireless interface comprising one or moreantennas, transmitters, receivers, transceivers, amplifiers, filters,control logic, and so forth.

In various aspects, the communications interface 10 may provide datacommunications functionality in accordance with a number of protocols.Examples of protocols may comprise various wireless local area network(WLAN) protocols, including the Institute of Electrical and ElectronicsEngineers (IEEE) 802.xx series of protocols, such as IEEE 802.11a/b/g/n,IEEE 802.16, IEEE 802.20, and so forth. Other examples of wirelessprotocols may comprise various wireless wide area network (WWAN)protocols, such as GSM cellular radiotelephone system protocols withGPRS, CDMA cellular radiotelephone communication systems with 1×RTT,EDGE systems, EV-DO systems, EV-DV systems, HSDPA systems, and so forth.Further examples of wireless protocols may comprise wireless personalarea network (PAN) protocols, such as an Infrared protocol, a protocolfrom the Bluetooth Special Interest Group (SIG) series of protocols(e.g., Bluetooth Specification versions 5.0, 6, 7, legacy Bluetoothprotocols, etc.) as well as one or more Bluetooth Profiles, and soforth. Yet another example of wireless protocols may comprise near-fieldcommunication techniques and protocols, such as electro-magneticinduction (EMI) techniques. An example of EMI techniques may comprisepassive or active radio-frequency identification (RFID) protocols anddevices. Other suitable protocols may comprise Ultra Wide Band (UWB),Digital Office (DO), Digital Home, Trusted Platform Module (TPM),ZigBee, and so forth.

In some embodiments, at least one non-transitory computer-readablestorage medium is provided having computer-executable instructionsembodied thereon, wherein, when executed by at least one processor, thecomputer-executable instructions cause the at least one processor toperform embodiments of the methods described herein. Thiscomputer-readable storage medium can be embodied in memory subsystem 8.

In some embodiments, the memory subsystem 8 may comprise anymachine-readable or computer-readable media capable of storing data,including both volatile/non-volatile memory and removable/non-removablememory. The memory subsystem 8 may comprise at least one non-volatilememory unit. The non-volatile memory unit is capable of storing one ormore software programs. The software programs may contain, for example,applications, user data, device data, and/or configuration data, orcombinations therefore, to name only a few. The software programs maycontain instructions executable by the various components of the system2.

In various aspects, the memory subsystem 8 may comprise anymachine-readable or computer-readable media capable of storing data,including both volatile/non-volatile memory and removable/non-removablememory. For example, memory may comprise read-only memory (ROM),random-access memory (RAM), dynamic RAM (DRAM), Double-Data-Rate DRAM(DDR-RAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM(PROM), erasable programmable ROM (EPROM), electrically erasableprogrammable ROM (EEPROM), flash memory (e.g., NOR or NAND flashmemory), content addressable memory (CAM), polymer memory (e.g.,ferroelectric polymer memory), phase-change memory (e.g., ovonicmemory), ferroelectric memory, silicon-oxide-nitride-oxide-silicon(SONOS) memory, disk memory (e.g., floppy disk, hard drive, opticaldisk, magnetic disk), or card (e.g., magnetic card, optical card), orany other type of media suitable for storing information.

In one embodiment, the memory subsystem 8 may contain an instructionset, in the form of a file for executing various methods, such asmethods including A/B testing and cache optimization, as describedherein. The instruction set may be stored in any acceptable form ofmachine readable instructions, including source code or variousappropriate programming languages. Some examples of programminglanguages that may be used to store the instruction set comprise, butare not limited to: Java, C, C++, C#, Python, Objective-C, Visual Basic,or .NET programming. In some embodiments a compiler or interpreter iscomprised to convert the instruction set into machine executable codefor execution by the processing subsystem 4.

FIG. 2 illustrates a network environment 20 configured to monitor andmaintain at least one deployed environment, in accordance with someembodiments. The network environment 20 includes a plurality of clusters22 a-22 c including at least one system configured to receive and/orprovide services, applications, systems, databases, and/or otherinfrastructure. For example, in some embodiments, each of the clusters22 a-22 c includes a cloud computing environment configured to provideone or more services, applications, systems, databases, etc. In someembodiments, each of the clusters 22 a-22 c may provide the sameenvironment and/or different resources or components.

Each of the clusters 22 a-22 c is in signal communication with amonitoring system 24. The monitoring system 24 is configured to monitorthe health and availability of one or more resources in each of theclusters 22 a-22 c. The monitoring system 24 may be configured tomonitor each cluster 22 a-22 c at any suitable level of granularity,such as, for example, general availability/up-time of each of theclusters 22 a-22 c, availability of specific resources within each ofthe clusters 22 a-22 c, and/or any other suitable granularity level. Insome embodiments, and as discussed in greater detail below, themonitoring system 24 is configured to implement a monitoring process toidentify, log correct, and/or generate notifications for active issueswith respect to at least one of the clusters 22 a-22 c.

In some embodiments, the monitoring system 24 is in signal communicationwith a self-healing system 26. The self-healing system 26 is configuredto implement one or more self-healing processes (or flows) to attempt tocorrect an active issue associated with an alert received by themonitoring system 24. In some embodiments, a received alert may belogged in a database cluster 40. One or more notifications may begenerated and provided to other systems, such as, for example, theadministrative system 28. Although embodiments are illustrated showingthe monitoring system 24, the self-healing system 26, and theadministration system 28 as independent systems, it will be appreciatedthat any of the illustrated systems may be combined and/or expanded intoone or more systems.

FIG. 3 illustrates a monitoring process 100 configured to be implementedby one or more systems, in accordance with some embodiments. Monitoringprocess 100 is configured to monitor the status of one or more resources102 in a deployed environment, such as, for example, within a cluster 22a-22 c. A local monitoring element 104 is executed by a systemassociated with the monitored environment, e.g., the cluster 22. Thelocal monitoring client is configured to identify one or more issuesexperienced by the resource 102 and generate an issue-specific alert106. For example, in various embodiments, the resource 102 may becomeunavailable, run out of memory, encounter an error, and/or otherwiseencounter an issue that causes the resource 102 to become partiallyand/or completely unusable.

When the local monitoring client 104 identifies an active issue relatedto the resource 102, an issue-specific alert 106 is generated. Theissue-specific alert 106 includes issue-specific information identifyingthe type of issue encountered by the resource 102. For example, invarious embodiments, the issue-specific alert 106 may indicate that theat least one resource 102 is down (i.e., unresponsive), low on diskspace, has connection issues, zookeeper issues, replication issues, CPU(central processing unit) issues, memory issues, and/or any other issuesand/or errors. In some embodiments, the issue-specific alert 106 may beresent (e.g., re-generated) at a predetermined interval, such as, forexample, a predetermined interval based on the type of alert 106, apredetermined interval based on the cluster 22 a-22 c at which the erroroccurred, and/or any other suitable interval. For example, in someembodiments, a resource down alert may be generated every 30 seconds, assuch an error may be considered a high priority error. In contracts, adisk-space alert, a zookeeper alert, a CPU issue alert, and/or a memoryissues alert may be considered a low priority alert and are generated ona longer interval, such as every 5 minutes. Other alerts, such as anactive connection alert or a replication issue alert may be consideredmedium-priority and may be generated at an interval between thelow-priority and high-priority intervals, such as, for example, every 60seconds.

In some embodiments, an alert normalization element 108 receives theissue-specific alert 106 and generates a normalized (or standardized)alert 110. The normalized alert 110 may include a container, such as,for example, a JSON container, configured to present the informationcontained within the issue-specific alert 106 in a standard format. Insome embodiments, the issue-specific alert 106 is wrapped in and/orotherwise included in the normalized alert 110.

The normalized alert 100 is provided to load balancing element 112configured to implement a load balancing process, such as the loadbalancing process 400 discussed below in conjunction with FIG. 6 . Insome embodiments, the load balancing element 112 assigns the receivednormalized alert 110 to one of a plurality of monitoring serversconfigured to implement an self-healing and notification process, suchas, for example, the self-healing and notification process 200 discussedin greater detail below with respect to FIG. 4 . The received normalizedalert 110 may be provided to an event logging element 114 configured tolog the normalized alert 110 and/or the active issue associated with thenormalized alert 110 in an event collection 118 maintained by a databasecluster, such as, for example, the database cluster 40.

After logging the normalized alert 110 in the event collection 118, anactive issue query 116 is performed to determine whether the normalizedalert 110 is associated with a previously identified active issue. Forexample, in some embodiments, an active issue collection 120 containingissue objects associated with active issues is queried to determinewhether an issue object exists for the issue associated with thenormalized alert 110. If the active issue was previously identified, theissue object associated with the active issue may be retrieved from theactive issue collection 120.

In some embodiments, a check 122 is performed to determine whether thenormalized alert 110 is the first alert that has been received for theassociated active issue. In some embodiments, the check 122 is combinedwith the active issue query 120 such that if the active issue query doesnot return an issue object, the normalized alert 110 is considered to bea first alert for the active issues. As an example, if a databasebecomes unresponsive (e.g., is down), an issue-specific alert 106 willbe generated indicating that the database is down. After receiving thenormalized alert 110 generated from the issue-specific alert, a system,such as the monitoring system 24, may query 116 an active issuescollection 120 to request an issue object for the active issues. If thequery 120 returns an issue object, at least one prior alert associatedwith the active issue has been received and the current alert is notconsidered a first alert. Alternatively, if an issue object is notreturned, then no prior alerts have been received and the current alertis the first alert.

If the check 122 determines that the received normalized alert 110 is afirst alert, an active issue generation element 124 generates a newissue object 126 associated with the active issue. The issue object 126may include any suitable data structure, such as, for example, adatabase document, container object, etc. In some embodiments, the issueobject 126 includes one or more fields configured to identify dataassociated with the active issue. For example, in various embodiments,the issue object 126 may include, but is not limited to, one or morefields identifying a server name, an alert type, alert information, aninitial alert time, a last updated time, a resolved time, an environmenttype, a critical alert indicator, a platform, etc. associated with theactive issue. The issue object 126 is stored in the active issuescollection 120.

If the check 122 determines that the received normalized alert 110 is asubsequent alert (e.g., is not the first alert), an active issue updatedelement 128 updates an existing active issue object 126. In someembodiments, the existing issue object 126 is updated to include thelatest alert time and information. In some embodiments, one or moreadditional processes, such as a process of automatically correcting adetected issue in a cloud environment discussed below, is initiated whenan active issue object 126 is created and/or updated.

FIG. 4 illustrates a self-healing and user-alerting process 200 for anactive issue, in accordance with some embodiments. As discussed above,in some embodiments, an active issue generating element 124 isconfigured to map the normalized alert 110 onto an issue object 126 a.In some embodiments, an alert type and platform are retrieved 204 froman alert type collection 206 and a platform information collection 207maintained by a database cluster, such as the database cluster 40. Insome embodiments, elements of the issue object 126 a are determined bythe retrieved alert type, a platform corresponding to the retrievedalert type, and/or any other suitable factor. For example, in oneembodiment, a specific type of alert and/or platform result in an issueobject 126 a including an alert type, an environment type, an initialtime (e.g., initial time of the alert/issue), a last updated element(e.g., time of last update regarding the issue), a platform element(e.g., platform executing the component having the issue), resolve timeelement (e.g., element indicating the time the issue was resolved), oneor more alert platform elements (e.g., indicating which platforms shouldreceive an alert based on the issue), an alert interval element (i.e.,indicating how often to generate the alerts), a platform event element(i.e., indicating which platforms should receive an event based on thealert), a self-healing element (i.e., indicating whether the particularalert type/active issue is amenable to self-healing execution), and/orany other suitable elements.

In some embodiments, the self-healing and alerting process 200 performsa self-healing check 208 to determine whether the active issueassociated with the issue object 126 a can be automatically corrected byone or more self-healing processes, for example, the self-healingprocess 300 discussed in greater detail below. In some embodiments, aself-healing element in the issue object 126 a is set to “yes” or “true”to indicate self-healing is possible and “no” or “false” to indicateself-healing is not possible. If the self-healing check 206 determinesthat self-healing is possible, the self-healing and alerting process 200attempts to automatically correct 210 the active issue using one or moreself-healing processes. A subsequent check 212 may be performed todetermine whether the attempt to correct 208 the active issue wassuccessful.

If the active issue is successfully corrected, the self-healing anduser-notification process 200 updates the resolved time 218 of the issueobject 126 a and closes 220 the active issue. For example, in someembodiments, a resolved time element is updated to the current time andan active issue element is transitioned to “false” after theself-healing process has completed. In some embodiments, the issueobject 126 a is migrated from the active issues collection 120 to aclosed issues collection 222 maintained by the database cluster 40.

If the active issue associated with the issue object 126 a cannot beself-healed, or if the attempt to automatically correct 210 the activeissue fails, one or more platform-specific notifications may begenerated. For example, in some embodiments, a check 214 is performed todetermine whether a first platform should be notified given the specificalert type, platform, and/or other parameters of the issue object 126 a.In some embodiments, the check 214 may be configured to determine thevalue of a first platform alert element included in the issue object 126a. If the first platform notification element is set to “yes” or “true,”the first platform is notified 216, for example, by generation of aplatform-specific notification for the first platform. Theplatform-specific notification may include information regarding theissue object, the resource associated with the active issue, the attemptto self-heal, and/or any other suitable information. In someembodiments, the first platform includes a chat platform, such as, forexample, slack.

In some embodiments, only a single alert is generated for each platform.For example, in some embodiments, a prior alert check is performed todetermine if an alert has already been generated for the first platform.If an alert has not been generated, the platform-specific alert isgenerated and the issue object 126 a may be updated, as discussed ingreater detail below, to include a first platform alert elementindicating that the first platform has been alerted. In someembodiments, the updated issue object 126 a is stored in the activeissue collection 120 of the database cluster 40.

Alternatively and/or simultaneously, a last updated check 224 may beperformed to determine if the last update received for the active issueis older than an update interval associated with the alert type. Forexample, in some embodiments, a last updated element in the issue object126 a is set to a time value corresponding to the last alert or updatereceived regarding the active issue associated with the component 102.An alert interval element of the issue object 126 a may be set to avalue corresponding to the frequency of an issue-specific alert 106generated for the active issue associated with the issue object 126 a.The value of the last updated element may be compared to the value inthe alert interval element. If the value of the last updated element isgreater than the value of the alert interval element, the active issueis considered resolved (as no additional alerts have been generated inthe alert interval). However, if the value of the last updated elementis less than the alert interval element, the self-healing anduser-alerting process 200 considers the issue associated with the issueobject 126 a to be active.

In some embodiments, if the last updated check 224 determines the issueremains active, one or more additional user notifications may begenerated. For example, in some embodiments, a production check 226 isperformed to determine if the component associated with the active issueis associated with a production environment. If the component isassociated with a production environment, i.e., the production check 226returns a true/yes result, a high-priority, platform-specificnotification 228 may be generated. The high-priority, platform-specificnotification may include a pager duty notification (e.g., a notificationgenerated and sent to a pager associated with an “on-call” individualfor urgent issues).

In some embodiments, an additional check is performed (not shown) todetermine if the active issue is a high-priority issue (e.g., isassociated with a high-priority or critical resource 102). For example,in some embodiments, a high-priority element of the issue object 126 amay be set to “true” when the issue object is related to a high-priorityresource 102 and/or cluster 22 a-22 c. In some embodiments, anadditional check may be performed to determine whether a high-prioritynotification has already been generated for the active issue. If ahigh-priority notification has not been previously generated, thehigh-priority notification 228 may be generated and transmitted to anotification platform, such as a pager platform. In some embodiments, ifany of the checks fail, a high-priority notification is not generated.

In some embodiments, additional notifications may be generated forspecific alert types, active issues, components, etc. For example, asecond platform-specific notification 232 may be generated and providedto a second platform, such as, for example, a JIRA platform for one ormore alert types. A second platform-specific notification check 230determines whether a second-platform specific notification 232 shouldbeen generated and provided to the second platform. For example, in someembodiments, if a second-platform notification element, such as a JIRAnotification element, is set to “true” or “yes,” the issue object 126 ais associated with an active issue that requires a notification to besent to a second platform. The second-platform notification 232 isprovided to the second platform. In some embodiments, thesecond-platform notification 232 is generated according to a processidentified by a second-platform event element included in the issueobject 126 a. In some embodiments, an additional check (not shown) maybe performed to determine if a prior notification has been generated andprovided to the second platform. If a prior second-platform-specificnotification has been generated, the self-healing and notificationprocess 200 may forego generation of the second-platform-specific alert.

In some embodiments, if a platform-specific notification is generated,the issue object 126 a is updated 234 to indicate that aplatform-specific alert has been generated. The updated issue object 126a is stored in the active issues collection 120. If one or moreadditional alerts are received for the same active issue, additionalnotifications are not generated for the platforms that have beenpreviously notified.

FIG. 5 illustrates automated healing process 300 for correction anactive issue, in accordance with some embodiments. In some embodiments,the automated healing process 300 may be implemented by one or moresystems, such as, for example, a monitoring system 24, a self-healingsystem 26, and/or any other suitable system. The self-healing process300 may be implemented independently and/or as part of an additionalprocess, such as, for example, as part of a self-healing andnotification process 200.

In some embodiments, a self-healing process is initiated as part of aself-healing and notification process 200 such as, for example, as anattempt to correct 208 an active issue, as discussed above inconjunction with FIG. 4 . An active self-healing collection 302maintained by the database cluster 40 is queried 304 to obtain aself-healing object (e.g., a self-healing document or database entry)associated with the identified active issue. A check 306 may beperformed on the result of the query 304. If a self-healing object hasalready been created, the check 306 is successful and the self-healingobject 304 is loaded from the self-healing collection 302. Theself-healing object may be updated 308 to include updated informationreceived from the latest alert that initiated the self-healing process300. In some embodiments, a last updated field of the self-healingobject is updated to include the current time, reflecting the mostrecent update.

If a self-healing object is not retrieved from the active self-healingcollection 302, the automated healing process 300 creates 310 an activeself-healing object in the active self-healing collection 302 maintainedin the database cluster 40. A self-healing flow may be loaded 312 basedon one or more aspects of a received normalized alert 110 and/or issueobject 126 a, such as, for example, an alert type of the issue-specificalert 106. For example, in some embodiments, a database down alert mayhave a first self-healing flow configured to re-enable the database anda disk space alert may have a second self-healing flow configured to addadditional disks to a cluster associated with the alert. It will beappreciated that any number of self-healing flows may be stored and/orretrieved. Each alert type may have a single self-healing flow, a sharedself-healing flow (e.g., shared by multiple alert types), and/ormultiple self-healing flows associated therewith. In some embodiments,the self-healing flows are stored in and/or retrieved from aself-healing flow collection 314 maintained by the database cluster 40,although it will be appreciated that the self-healing flows may bestored in and/or retrieved from any suitable storage location.

After retrieving 312 the self-healing flow, the automated healingprocess 300 executes 316 the self-healing flow and attempts to correctthe underlying active issue. After executing the self-healing flow, acheck 318 may be performed to determine whether the self-healing flowwas successful. For example, the check 318 may include polling orquerying the component 102 associated with the active issue, waiting apredetermined time period greater than the alerting period to determineif additional alerts are generated, and/or any other suitableconfirmation process.

If the self-healing flow successfully corrected the active issue, theself-healing object is migrated 320 (e.g., moved) from the activeself-healing collection 302 to a closed self-healing collection 322maintained by the database cluster 40. If the self-healing flow wasunsuccessful, the self-healing object may be updated 324 to indicate theself-healing status is “failed.” In some embodiments, after updating 324the self-healing status, additional alerting processes as discussedabove with respect to FIG. 4 are triggered. In some embodiments, theupdated self-healing object is stored in the active self-healingcollection 302 such that the self-healing object may be loaded from theactive self-healing object 302 by during future attempts to correct theactive issue.

FIG. 6 illustrates a backend logging process 400 for logging alertsgenerated by a cloud monitoring process, in accordance with someembodiments. The backend logging process 400 may be executed as part ofa monitoring process, such as the cloud process 200 discussed above,and/or may be executed as a stand-alone process. The backend loggingprocess 400 may be implemented by any suitable system, such as, forexample, a monitoring system 24, a self-healing system 26, and/or anyother suitable system.

In some embodiments, a normalized alert 110 is received and provided toa load balancer 402 configured to implement load balancing for amonitoring process, such as the monitoring process 200 discussed above.The load balancer 402 selects one of a plurality of servers 404 a-404 cto receive and process the normalized alert 110. In some embodiments,the server 404 a-404 c is selected based on one or more load balancingprocesses. After the normalized alert 110 is provided to the selectedone of the plurality of servers 404 a-404 c, an alert processing element406 a-406 c is configured to process the normalized alert 110. The alertprocessing element 406 a-406 c may be configured to process specificalerts having a predetermined format, such as a JSON package, generatedby the alerting process 100, as discussed in greater detail above. Insome embodiments, processing may include, but is not limited to,generating an issue object based on the normalized alert 110.

After processing the alert, a logging element 408 is configured to logthe normalized alert 110 (and/or the active issue associated with thenormalized alert 110) in a database cluster 40. The database cluster 40may include the same database cluster 40 illustrated in FIG. 3 and/ormay be a different database cluster. Each normalized alert 110 may bestored in one of a plurality of nodes 410 a-410 c maintained within thedatabase cluster 40.

Although the subject matter has been described in terms of exemplaryembodiments, it is not limited thereto. Rather, the appended claimsshould be construed broadly, to include other variants and embodiments,which may be made by those skilled in the art.

What is claimed is:
 1. A system comprising: one or more processors; anda memory resource storing a set of instructions that when executed bythe one or more processors, cause the one or more processors to: basedon an alert of an active issue of a system resource, identify a firstissue object associated with the alert by making a first set ofdeterminations; based on the first issue object, determine whether thealert is a first alert or a subsequent alert; when the alert is thefirst alert, based on the first issue object, determine whether theactive issue associated with the first issue object can be automaticallycorrected by a first self-healing process; when the alert is asubsequent alert, based on the first issue object and the subsequentalert, determine whether the active issue associated with the firstissue object can be automatically corrected by a second self-healingprocess; based on determining that the active issue associated with thefirst issue object can be automatically corrected by the firstself-healing processes, implement the first self-healing process; andbased on determining the active issue associated with the first issueobject can be automatically corrected by the second self-healingprocesses, implement the second self-healing process; wherein the firstissue object includes a resolved time element, and execution of the setof instructions, by the one or more processors, further causes the oneor more processors to: determine the first self-healing process or thesecond self-healing process was successful or failed; responsive todetermining that the first self-healing process or the secondself-healing process was successful, update the resolved time element ofthe first issue object; and responsive to determining the firstself-healing process or the second self-healing process failed,implement one or more platform alert processes.
 2. The system of claim1, wherein the first issue object includes a self-healing element, theself-healing element indicates whether the active issue associated withthe first issue object can be automatically corrected by one or moreself-healing processes.
 3. The system of claim 2, wherein theself-healing element indicates that the active issue of the first issueobject can be automatically corrected.
 4. The system of claim 1, whereinthe first issue object includes a platform alert element, and whereinthe one or more platform alert processes includes: based at least on theplatform alert element of the first issue object, determining whether aparticular platform should be notified.
 5. The system of claim 4,wherein the one or more platform alert processes further includes: basedon determining that the particular platform should be notified,generating a platform-specific notification for the particular platform.6. The system of claim 5, wherein the platform-specific notificationincludes information regarding the first issue object, the systemresource associated with the first issue object, and the attempt toself-heal.
 7. A computer-implemented method comprising: based on analert of an active issue of a system resource, identifying a first issueobject associated with the alert by making a first set ofdeterminations; based on the first issue object, determine whether thealert is a first alert or a subsequent alert; when the alert is thefirst alert, based on the first issue object, determining whether theactive issue associated with the first issue object can be automaticallycorrected by a first self-healing process; when the alert is asubsequent alert, based on the first issue object and the subsequentalert, determining whether the active issue associated with the firstissue object can be automatically corrected by a second self-healingprocess; based on determining that the active issue associated with thefirst issue object can be automatically corrected by the firstself-healing process, implementing the first self-healing process; andbased on determining the active issue associated with the first issueobject can be automatically corrected by the second self-healingprocesses, implementing the second self-healing process; wherein thefirst issue object includes a resolved time element, and once the one ormore self-healing processes has been implemented; determining the firstself-healing process or the second self-healing process was successfulor failed; responsive to determining that the first self-healing processor the second self-healing process was successful, updating the resolvedtime element of the first issue object; and responsive to determiningthe first self-healing process or the second self-healing processfailed, implementing one or more platform alert processes.
 8. Thecomputer-implemented method of claim 7, wherein the first issue objectincludes a self-healing element, the self-healing element indicateswhether the active issue associated with the first issue object can beautomatically corrected by one or more self-healing processes.
 9. Thecomputer-implemented method of claim 8, wherein the self-healing elementindicates that the active issue of the first issue object can beautomatically corrected.
 10. The computer-implemented method of claim 7,wherein the first issue object includes a platform alert element, andwherein the one or more platform alert processes includes: based atleast on the platform alert element of the first issue object,determining whether a particular platform should be notified.
 11. Thecomputer-implemented method of claim 10, wherein the one or moreplatform alert processes further includes: based on determining that theparticular platform should be notified, generating a platform-specificnotification for the particular platform.
 12. The computer-implementedmethod of claim 11, wherein the platform-specific notification includesinformation regarding the first issue object, the system resourceassociated with the first issue object, and the attempt to self-heal.13. A non-transitory computer-readable medium storing instructions, thatwhen executed by one or more processors of a system, cause the systemto: based on an alert of an active issue of a system resource, identifya first issue object associated with the alert by making a first set ofdeterminations; based on the first issue object, determine whether thealert is a first alert or a subsequent alert; when the alert is thefirst alert, based on the first issue object, determine whether theactive issue associated with the first issue object can be automaticallycorrected by a first self-healing process; when the alert is asubsequent alert, based on the first issue object and the subsequentalert, determine whether the active issue associated with the firstissue object can be automatically corrected by a second self-healingprocess; based on determining that the active issue associated with thefirst issue object can be automatically corrected by the firstself-healing process, implementing the first self-healing process; andbased on determining the active issue associated with the first issueobject can be automatically corrected by the second self-healingprocesses, implement the second self-healing process; wherein the firstissue object includes a resolved time element, and execution of theinstructions, by the one or more processors, further causes the systemto: determining the first self-healing process or the secondself-healing process was successful or failed; responsive to determiningthat the first self-healing process or the second self-healing processwas successful, update the resolved time element of the first issueobject; and responsive to determining the first self-healing process orthe second self-healing process failed, implementing one or moreplatform alert processes.
 14. The non-transitory computer-readablemedium of claim 13, wherein the first issue object includes aself-healing element, the self-healing element indicates whether theactive issue associated with the first issue object can be automaticallycorrected by one or more self-healing processes.
 15. The non-transitorycomputer-readable medium of claim 14, wherein the self-healing elementindicates that the active issue of the first issue object can beautomatically corrected.